Method and system for evacuation route guidance of occupants using augmented reality of mobile devices

ABSTRACT

A system for guiding an evacuation route using augmented reality of mobile devices according to an embodiment of the present disclosure comprises an occupant&#39;s mobile device installed with an evacuation guidance application for showing an evacuation route from a current location of the occupant to an evacuation point in the occurrence of a disaster; and a server providing a plurality of evacuation route information to the mobile device, wherein the mobile device executes the evacuation guidance application to display an evacuation guidance interface, displays a captured landscape image on an image display area of the evacuation guidance interface, displays one of the plurality of evacuation routes set based on specific information related to the type of disaster, the user&#39;s exercise ability, and personal conditions on a map display area of the evacuation guidance interface and displays guide information for displaying a movement direction from a current location along an evacuation route indicated on the map display area by superimposing the guide information on the image display area together with the landscape image.

TECHNICAL FIELD

The present disclosure relates to a method and a system for evacuationguidance of occupants using augmented reality of mobile devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Korean Patent ApplicationsNO 10-2021-0152855 filed on Nov. 9, 2021, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND

Along with the rapid industrial development, advances of society, andincrease of the population, medium- and large-scale buildings such ascultural, commercial, and residential facilities grow in size andprovide multi-purpose uses. Accordingly, the inside of a building iscomplex, and users often encounter difficulty finding a route inside thebuilding. These features of today's buildings may exacerbate theconfusion to the activity of evacuating and rescuing users of thecorresponding building, particularly in the event of a disaster such asfire, gas leak, earthquake, or terror attack. Therefore, there hasarisen a need for an evacuation route guidance system that evacuees maycount on in a disaster situation inside a complex, medium- orlarge-scale building. Recently, as evacuation-related technologies haveadvanced in various fields, it has become possible to design anevacuation route guidance system that more actively responds to disastersituations. The evacuation-related technologies currently developedinclude 5 communication technology that determines the indoor locationof an evacuee, sensor technology that senses a disaster, sensortechnology that evaluates the risk of building collapse, and technologythat monitors the spread of a disaster. In addition, by integratingBuilding Information Modeling (BIM)-based smart construction forbuilding facility management with the evacuation-related technologycurrently developed, the situation inside a building may becomprehended. This situation-awareness allows designing a new evacuationroute guidance system. The prior art focuses on predicting a disastersituation indoors; therefore, the prior art has difficulty providing anevacuation route suitable for evacuees in a chaotic situation such as adisaster situation. Recently, to solve this problem, Korean registeredpatent No. 10-2124097 disclosed a method that detects smoke or flamesusing an IoT sensor and guides an evacuation route in real-time throughan occupant's smartphone. However, considering dynamically changingdisaster situations at various points on the evacuation route, anerroneous operation or damage of the IoT sensors due to fire or the likereadily incapacitates the disclosed method in guiding an appropriateevacuation route. Moreover, Korean laid-open patent publication No.10-2019-0138222 discloses a method for guiding an evacuation route in abuilding based on augmented reality through a smartphone. However, thedisclosed method provides only a preset evacuation route and is not ableto respond appropriately to an unexpected situation on the evacuationroute by checking an unexpected situation in advance.

PRIOR ART REFERENCES Patents

-   (Patent 001) Korean registered patent publication No. 10-2124097-   (Patent 002) Korean laid-open patent publication No. 10-2019-0138222-   (Patent 003) European registered patent publication No. 3149716

SUMMARY

The present disclosure provides a method and a system for guiding anevacuation route for occupants using augmented reality of mobiledevices, which enable the occupants in a building to evacuate quicklyand safely in a disaster situation.

Also, the present disclosure provides a method and a system for guidingan evacuation route for occupants using augmented reality of mobiledevices, which enable the occupants to evacuate through an alternativeroute by identifying an unexpected situation at a plurality of points onthe evacuation route.

Also, the present disclosure provides a method and a system for guidingan evacuation route for occupants using augmented reality of mobiledevices, which enable the occupants to evacuate safely by providinglocation information of various emergency items suitable for the type ofdisaster on the way of the evacuation and information on how to use theemergency items.

Also, the present disclosure provides a method and a system for guidingan evacuation route for occupants using augmented reality of mobiledevices, which help evacuees quickly determine whether to evacuatethrough an alternative route by analyzing captured images of variouspoints on the evacuation route using deep learning-based machinelearning and determining the accessibility of the analyzed points forthe evacuees.

Also, the present disclosure provides a method and a system for guidingan evacuation route for occupants using augmented reality of mobiledevices, which enable evacuees with disabilities or children to receiveinformation related to evacuation correctly by utilizing a plurality ofevacuation guidance assisting devices installed on the ceiling orpassageways inside a building and guiding evacuation to an emergencyexit or an escape place in a disaster situation such as fire orearthquake, where messages are output audibly or visually by reflectingthe situation specific to individuals or the current environmentalconditions.

A system for guiding an evacuation route using augmented reality ofmobile devices according to an embodiment of the present disclosure maycomprise an occupant's mobile device installed with an evacuationguidance application for showing an evacuation route from a currentlocation of the occupant to an evacuation point in the occurrence of adisaster; and a server providing a plurality of evacuation routeinformation to the mobile device, wherein the mobile device executes theevacuation guidance application to display an evacuation guidanceinterface; displays a captured landscape image on an image display areaof the evacuation guidance interface; displays one of the plurality ofevacuation routes set based on specific information related to the typeof disaster, the user's exercise ability, and personal conditions on amap display area of the evacuation guidance interface; and displaysguide information for displaying a movement direction from a currentlocation along an evacuation route indicated on the map display area bysuperimposing the guide information on the image display area togetherwith the landscape image.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the server prioritizes the plurality of evacuation routes basedon specific information related to the type of disaster, the user'sexercise ability, and personal conditions; and the mobile devicedisplays one of the evacuation routes selected based on the prioritiesof the plurality of evacuation routes received from the server on themap display area.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the mobile device receives location information of at least oneemergency item related to the type of disaster from the server anddisplays the location information of an evacuation route on the mapdisplay area.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein, when the mobile device is located at a point on an evacuationroute displayed on the map display area that matches the location atwhich the emergency item is disposed, the mobile device analyzes acaptured image to determine whether an emergency item is detected withinthe captured image; displays an indication to facilitate identificationof the emergency item when the emergency item is detected within thecaptured image; and displays a tag including information on how to usethe detected emergency item.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein, when the mobile device is located at a point on an evacuationroute displayed on the map display area that matches the location wherethe emergency item is disposed, the mobile device analyzes a capturedimage to determine whether an emergency item is detected in the capturedimage and transmits use completion information of the emergency item andlocation information of the emergency item completed for use to theserver when the emergency item is not detected in the captured image;and to remove the location of an emergency item completed for usedisplayed on the evacuation route, the server transmits information ofthe emergency item completed for use and location information of theemergency item completed for use to the mobile devices of all occupantscommunicating with the server.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the mobile device receives waypoint state images, which areimages captured by other mobile devices at a plurality of differentpoints on the remaining route from a current location to an evacuationpoint, from the server and displays the received images.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the mobile device reconfigures an evacuation route that detoursa point on a waypoint state image selected in response to a selection ofthe displayed waypoint state image and a route reconfigurationaffordance and displays the reconfigured evacuation route on the mapdisplay area.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein waypoint state images displayed on the mobile device are updatedperiodically according to a location change of the mobile device.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the number of waypoint state images transmitted from the serverto the user's mobile device is determined based on the remainingdistance from a current location of the mobile device to the evacuationpoint, the number of other users' mobile devices on the remaining routefrom the current location to the evacuation point, and separationdistances among positions of other users' mobile devices.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the server receives captured image data for the same pointphotographed by each of a plurality of users moving along the sameevacuation route, analyzes the plurality of captured image data todetect environmental state information of the same point, and, if thesame point is determined as an avoidance point through which movement isnot appropriate, transmits a request for reconfiguring an evacuationroute, location information of the avoidance point, and images capturingthe avoidance point to the mobile devices of a plurality of users movingalong the same evacuation route.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devicesfurther comprises an evacuation guidance assisting device paired with atleast one mobile device located within a preconfigured radius, whereinthe evacuation guidance assisting device receives specific informationrelated to the type of disaster, the user's exercise ability, andpersonal conditions from the paired mobile device and provides audibleand/or visual messages suitable for the user's conditions and a currentsituation.

In another aspect, the present disclosure may provide a system forguiding an evacuation route using augmented reality of mobile devices,wherein the mobile device calculates an average movement speed over apreconfigured distance and an amount of battery consumption of themobile device consumed while moving the preconfigured distance bydetermining whether the movement of the preconfigured distance along theevacuation route has been completed, determines whether an estimatedamount of battery consumption required for movement along the remainingroute exceeds the amount of current battery charge, and executes areal-time disaster situation notification mode and a disaster situationbattery consumption control mode if the estimate does not exceed theamount of current battery charge; the mobile device displays a messengerapplication having use records for a predetermined period and displaysinformation on main conversation participants who have conducted aconversation for a preconfigured time by extracting the information inthe messenger application, in response to selection of the displayedinformation on main conversation participants who have conducted aconversation, transmits real-time evacuation route information andlocation information of a current user to the selected main conversationparticipants, and executes a conversation mode with the selected mainconversation participants if a response is received from the selectedmain conversation participants within a preconfigured time; and themobile device determines whether biometric information of apre-registered user is detected, maintains a display of the mobiledevice to be in a turned-on state when the biometric information isdetected, transmits an image photographing the user and captured imageof the front scene to the selected main conversation participants, andturns off the display and provides an audio guide of evacuationinformation while the biometric information is not detected.

The embodiment according to the present disclosure may provide a methodand a system for guiding an evacuation route for occupants usingaugmented reality of mobile devices, which enable occupants locatedinside a building to evacuate quickly and safely in a disastersituation.

The embodiment according to the present disclosure may provide a methodand a system for guiding an evacuation route for occupants usingaugmented reality of mobile devices, which enable occupants to evacuatevia an alternative route by recognizing unexpected situations at aplurality of points on then evacuation route.

The embodiment according to the present disclosure may solve a problemof wasting time to collect already used emergency items in an emergencysituation requiring a prompt response by providing location informationof various emergency items suitable for the type of disaster located onan evacuation route and information on how to use the emergency items tothe occupants so that the occupants may quickly recognize the locationsof the emergency items during evacuation and by updating the locationinformation of the emergency items according to whether the emergencyitems are in use or not.

The embodiment according to the present disclosure may determine aplurality of evacuation routes and their priority by considering thetype of disaster, the user's exercise ability, and personal conditionsand provide an evacuation route suitable for current circumstances andthe user.

The embodiment according to the present disclosure enables a user torecognize actual situation information of a plurality of points on theremaining route through images while moving along an evacuation route,thus enabling the user to make a prompt decision on the currentsituation and accessibility of the points to be passed and enables theuser to determine an alternative evacuation route that avoids pointsexpected to be impossible to move through.

The embodiment according to the present disclosure analyzes a pluralityof captured images photographing particular points on the sameevacuation route using deep learning-based machine learning to determinethe accessibility of the analyzed points and informs the user of theanalysis result, thus enabling the user to quickly determine the degreeof safety of a current evacuation route and a need for reconfiguring theevacuation route.

The embodiment according to the present disclosure enables evacuees withdisabilities or children to receive information related to evacuationcorrectly by outputting messages relevant to the current circumstancesand the user's specific situation through an evacuation guidanceassisting device paired with the user's mobile device based on specificinformation related to the type of disaster, the user's exerciseability, and personal conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the structure for implementing anevacuation guidance system according to one embodiment of the presentdisclosure.

FIG. 2 , FIG. 3 , FIG. 4 , FIG. 5 to FIG. 6 illustrate variousevacuation guidance interfaces displayed on a mobile device for helpinga user move to an evacuation point safely along an evacuation routeafter an evacuation guidance application is executed on the mobiledevice.

FIG. 7 illustrates a method for displaying waypoint state images.

FIG. 8 illustrates an evacuation guidance interface related to a methodfor reconfiguring an evacuation route.

FIG. 9 illustrates a system according to an embodiment for describing amethod for a server to receive image data from a plurality of mobiledevices and analyze environmental state information.

FIG. 10 illustrates an evacuation guidance interface related todisplaying avoidance points and guidance tags for reconfiguring a route.

FIG. 11 illustrates an evacuation guidance interface for describing amethod for detecting a state in which an emergency item has been removedand removing location information of the emergency item from anevacuation route.

FIG. 12 , FIG. 13 and FIG. 14 is a flow diagram illustrating executionof a real-time disaster situation notification mode and a disastersituation battery consumption control mode according to a method forguiding an evacuation route for occupants using augmented reality ofmobile devices.

FIG. 15 and FIG. 16 illustrate an evacuation guidance interfaceaccording to execution of a real-time disaster situation notificationmode.

DETAILED DESCRIPTION

Since the present disclosure may be modified in various ways and mayprovide various embodiments, specific embodiments will be depicted inthe appended drawings and described in detail with reference to thedrawings. The effects and characteristics of the present disclosure anda method for achieving them will be clearly understood by referring tothe embodiments described later in detail together with the appendeddrawings. However, it should be noted that the present disclosure is notlimited to the embodiment disclosed below but may be implemented invarious forms. In the following embodiments, the terms such as first andsecond are introduced to distinguish one element from the others, andthus the technical scope of the present disclosure should not be limitedby those terms. Also, a singular expression should be understood toindicate a plural expression unless otherwise explicitly stated. Theterm include or have is used to indicate existence of an embodiedfeature or constituting element in the present disclosure; and shouldnot be understood to preclude the possibility of adding one or moreother features or constituting elements. Also, constituting elements inthe figure may be exaggerated or shrunk for the convenience ofdescriptions. For example, since the size and thickness of each elementin the figure has been arbitrarily modified for the convenience ofdescriptions, it should be noted that the present disclosure is notnecessarily limited to what has been shown in the figure.

In what follows, embodiments of the present disclosure will be describedin detail with reference to appended drawings. Throughout thedisclosure, the same or corresponding constituting element is assignedthe same reference number, and repeated descriptions thereof will beomitted.

FIG. 1 is a block diagram illustrating the structure for implementing anevacuation guidance system according to one embodiment of the presentdisclosure.

An evacuation guidance system 10 according to one embodiment maycomprise a mobile device 100 and a server 200. According to variousembodiments, the evacuation guidance system 10 may comprise a mobiledevice 100, a server 200, and an evacuation guidance assisting device300.

The mobile device 100 and the server 200 are connected to each otherthrough a communication line such as the Internet.

For example, the mobile device 100 may be a mobile terminal 20 such as asmartphone. The mobile device 100 may be a wearable computing devicesuch as smart glasses 30 or a head-mounted display in variousembodiments. The smart glasses 30 type mobile device 100 may have adisplay system that includes glasses transmitting the light for the userto see the surrounding physical space while the user wears the glassesand displaying virtual content (in the embodiment, various types ofaugmented reality-based information) in the user's field of view. Themobile device 100 may receive the user input. The mobile device 100 ofthe mobile terminal 20 type may include a physical button capable ofreceiving a physical input, a touch display capable of receiving a touchinput, and a microphone system capable of receiving a voice command.Other smart glasses 30 type mobile devices 100 may include a gesturerecognition system capable of recognizing a gesture from a user's handgesture. The user may input a command signal to the smart glasses 30through a gesture.

An evacuation guidance application 100 a is installed in the mobiledevice 100 to realize the evacuation guidance system 10. In variousembodiments, in response to knowing that the mobile device 100 islocated in a predetermined building, a message regarding whether todownload and install the evacuation guidance application 100 a may betransmitted to the mobile device 100. The user may download and installthe evacuation guidance application 100 a on the mobile device 100 usingthe information or a link in the message.

The evacuation guidance application 100 a is software for presenting anevacuation route from the current location of a user (occupant) to anevacuation point when a disaster such as fire or earthquake occurs in abuilding.

The user may start the evacuation guidance application 100 a by apredetermined operation. By installing the evacuation guidanceapplication 100 a on the mobile device 100, the user may quickly arriveat a safe place such as an evacuation point even when the userencounters a sudden disaster.

The mobile device 100 includes at least one processor, a memory, acamera, a display device, various sensors, and a communication device.The processor provides the function of the evacuation guidanceapplication 100 a to the user by executing the evacuation guidanceprogram stored in the memory. According to the execution of theevacuation guidance application 100 a, the camera is driven tophotograph the landscape.

The mobile device 100 may receive information on a plurality ofevacuation routes from the server 200 and display an evacuation routesuitable for the user through the display device.

FIGS. 2 to 6 illustrate various evacuation guidance interfaces displayedon a mobile device for helping a user move to an evacuation point safelyalong an evacuation route after an evacuation guidance application isexecuted on the mobile device.

Referring to FIG. 2 , when the user executes the evacuation guidanceapplication 100 a on the mobile device 100, the evacuation guidanceinterface 110 may be displayed. After that, the camera of the mobiledevice 100 may start photographing, and the photographed landscape imagemay be displayed through the display device. In addition, the mobiledevice 100 may display a map including the user's current location.

The user's current location may be the location of the mobile device 100carried by the user. Various methods may determine the indoor locationof the mobile device 100, including a position estimation method byanalyzing sensing information of a geomagnetic sensor embedded in themobile device 100, a Wi-Fi positioning method for estimating thelocation using surrounding Wi-Fi signals, a Wi-Fi/Bluetooth basedfingerprinting-based positioning method belonging to theWi-Fi/Bluetooth-based indoor positioning technique, atrilateration-based positioning method, and a positioning method usingcameras. However, the present disclosure is not limited to the positionestimation methods described above.

The captured landscape image is displayed on the image display area 111,and a map is displayed on the map display area 112. The image displayarea 111 and the map display area 112 may be displayed separately fromeach other. The figure illustrates an example in which the mobile device100 is a mobile terminal 20. However, the present disclosure is notlimited to the specific example. In the case of smart glasses 30, theimage display area 111 and the map display area 112 may be displayed atdifferent positions in various forms.

The map display area 112 may display current location information cl ofthe mobile device 100. Also, the location information cl may be changedon a map displayed on the map display area 112 according to a locationchange of the mobile device 100. In addition, the location informationcl may move in response to the movement of the mobile device 100.

Referring to FIGS. 2 and 3 , the evacuation guidance interface 110 maydisplay a plurality of state input affordances ia1, ia2, and ia3 bysuperimposing them on a partial area of the landscape image displayed onthe image display region 111. The evacuation guidance interface 110 mayrespond to selection (for example, a touch input) of one of thedisplayed state input affordances ia1, ia2, and ia3 and display aplurality of detailed information input affordances sa matching theselected affordance ia1.

One of the plurality of state input affordances ia1, ia2, and ia3 mayoperate as a button that executes displaying the detailed informationinput affordance for inputting the type of disaster. Another one of theplurality of state input affordances ia1, ia2, and ia3 may operate as abutton that executes displaying the detailed information inputaffordance related to the user's exercise ability. Yet another one ofthe plurality of state input affordances ia1, ia2, and ia3 may operateas a button that executes displaying the detailed information inputaffordance related to the personal conditions of the user.

The detailed information input affordance for inputting the type ofdisaster may be used as a button for inputting information such asearthquake, fire, or hazardous material leakage.

The detailed information input affordance related to the user's exerciseability may be used as a button for inputting information such as “Icannot walk, I cannot climb, I cannot run, or I have my leg injured.” Inaddition, the detailed information input affordance related to personalconditions of the user may be used as a button for inputting informationsuch as “children under a certain age, a hearing-impaired person, anaccompanying child, a wheelchair, or a stroller.” The user may inputspecific information related to the type of the current disaster, theuser's current exercise ability, and personal conditions by selectingone of the plurality of displayed detailed information input affordancessa.

The mobile device 100 may transmit the input specific informationrelated to the type of disaster, the user's exercise ability, andpersonal conditions and current location information of the mobiledevice 100 to the server 200.

The server 200 may determine a plurality of evacuation routes from thecurrent location of the mobile device 100 to an evacuation point basedon the received specific information related to the type of disaster,the user's exercise ability, and personal conditions; and the locationinformation of the mobile device 100.

The server 200 may determine an appropriate evacuation route based onthe specific information related to the type of disaster, the user'sexercise ability, and personal conditions.

For example, the server 200 may determine an optimal evacuation route toan available evacuation point in the occurrence of fire among evacuationpoints registered in advance. In addition, if the user has a leg injury,is moving in a wheelchair, or is pushing a stroller, the server 200determines that the user is unable to use stairs and determines anevacuation route that avoids the stairs. In addition, if the server 200determines that it is necessary to provide additional evacuationguidance assistance information to the user, such as when the user is achild under a specific age or a hearing-impaired person, the server 200may determine an evacuation route that passes an area in which anevacuation guidance assisting device 300 is located.

The server 200 may generate a plurality of evacuation route informationand determine their priorities. The server 200 may determine thepriority of the plurality of evacuation route information based onspecific information related to the type of disaster, the user'sexercise ability, and personal conditions.

The server 200 may provide the plurality of determined evacuation routeinformation to the mobile device 100.

The evacuation guidance interface 110 displays an evacuation route withthe highest priority among the plurality of evacuation routes receivedfrom the server 200 in response to the selection of the displayed routereconfiguration affordance ra by superimposing the evacuation route onthe map displayed in advance on the map display area 112.

In various embodiments, when the evacuation guidance interface 110receives the selection of the route reconfiguration affordance ra againafter displaying an evacuation route, the evacuation guidance interface110 may display another evacuation route of the second-highest priorityamong the evacuation routes displayed. Accordingly, the user may changethe displayed evacuation route by selecting the route reconfigurationaffordance ra.

In various embodiments, the evacuation guidance interface 110 mayadditionally display priority information of the displayed evacuationroutes.

Referring to FIG. 4 , the evacuation guidance interface 110 may displaymovement direction guide information superimposed on the landscape imagein the image display area 111 based on the location information of themobile device 100 and the inclination angle information of the mobiledevice 100.

For example, the evacuation guidance interface 110 may guide a movementdirection for the user by displaying a route to be followed from thecurrent location of the mobile device 100 along an evacuation routeusing a means such as an arrow symbol.

The evacuation guidance interface 110 may display the image captured bythe mobile device 100 on the image display area 111 and superimpose amovement direction to be followed from the current location along anevacuation route and evacuation route guide information on the movementdistance on the displayed image. The mobile device 100 guides the userto move in the correct direction along the evacuation route byappropriately changing the evacuation route guide information based onthe location and inclination angle of the mobile device 100.

Referring to FIG. 5 , the evacuation guidance interface 110 may displaylocation information of at least one emergency item located on theevacuation route on the map display area 112. In various embodiments,when the evacuation route information displayed on the map display area112 is confirmed, the mobile device 100 may receive, from the server200, location information of the emergency items located on theevacuation route to be displayed. Also, when the evacuation routeinformation displayed on the map display area 112 is changed, the mobiledevice 100 may receive location information of the emergency itemslocated on the changed evacuation route from the server 200.

For example, when the type of disaster received is fire, the evacuationguidance interface 110 may display, on the map display area 112, onlythe emergency items located on the evacuation route displayed on the mapdisplay area 112 by filtering location information of a plurality ofemergency items on the map received from the server 200.

In various embodiments, the evacuation guidance interface 110 maydisplay an icon esi of an emergency item in a form easy for identifyingthe type of emergency item at the location of the correspondingemergency item on the map display area 112.

If the evacuation guidance interface 110 determines that the mobiledevice 100 is located at a point of a route matching the location wherean emergency item is disposed, the evacuation guidance interface 100 mayanalyze captured images to determine whether the emergency item isdetected in the captured images. When the mobile device 100 detects anemergency item in the captured images, the evacuation guidance interface110 may perform various additional displays to help the user quicklyidentify the emergency item es (according to the illustrated example, arectangular bounding box enclosing the emergency item may be displayed).Also, the evacuation guidance interface 110 may display a tag tgincluding information on how to use the emergency item es on the imagedisplay area 111. The tag tg may be displayed around the detectedemergency item es.

Referring to FIG. 6 , the server 200 may receive, from the mobile device100, evacuation route information currently displayed on the mobiledevice 100. In addition, the server 200 may periodically receive, fromthe mobile device 100, the current location of the mobile device 100.Also, the server 200 may periodically receive from the mobile device 100an image captured by the mobile device 100 and information on thelocation at which the mobile device 100 captures the image. In variousembodiments, for fast data transmission and reception, an image may betransmitted from the mobile device 100 to the server 200 in the form ofimage data.

The server 200 may transmit waypoint state image data to the mobiledevice 100, which is the image information captured by other users'mobile devices at a plurality of different points on the evacuationroute from the current location of the mobile device 100 to theevacuation point. In various embodiments, the server 200 may transmitthe captured image to the mobile device 100 in the form of image data.

FIG. 7 illustrates a method for displaying waypoint state images.

Referring to FIGS. 6 and 7 by collecting image data taken through aplurality of mobile devices on the evacuation route, the server 200 mayreceive, over the evacuation route p from a start point SP to anevacuation point SH, images a, b, c captured by other users' mobiledevices 100 ap 1, 100 ap 2, 100 ap 3 located at different points on theroute from the current location of the user's mobile device 100 p to theevacuation point SH. Then, the server 200 may transmit the waypointstate image data, which is the received image a, b, c, to the user'smobile device 100 p. According to the illustrated example, the server200 transmits images received from three mobile devices 100 ap 1, 100 ap2, and 100 ap 3 of other users to the user's mobile device 100 p.However, the present disclosure is not limited to the example. Thenumber of images transmitted from the server 200 to the user's mobiledevice 100 p may be changed based on the remaining distance from thecurrent location of the user's mobile device 100 p to the evacuationpoint SH, the number of other users' mobile devices on the remainingroute from the current location to the evacuation point, and separationdistances among positions of other users' mobile devices.

The mobile device 100 may receive the waypoint state image data from theserver 200 and display the received image on the mobile device 100.According to various embodiments, the map display area 112 may displaythe location where each of the plurality of waypoint state images hasbeen captured.

The user may check the images of a plurality of points on the remainingevacuation route on the mobile device 100. In addition, images for aplurality of points on the remaining route may be periodically updatedas the location of the mobile device 100 is changed. The user may check,through the images, the information on the states of people at aplurality of points evacuating along the remaining route, thesurrounding environment, and whether or not they are safe. When the userdetermines that one point of the evacuation route is inappropriate, theuser may change the displayed evacuation route by selecting the routereconfiguration affordance ra to reconfigure an evacuation route andmove through another evacuation route.

FIG. 8 illustrates an evacuation guidance interface related to a methodfor reconfiguring an evacuation route.

Referring to FIG. 8 , in various embodiments, the evacuation guidanceapplication 100 a may change the evacuation route and display anevacuation route cp detouring a selected point on the image in responseto the selection of one of the images of a plurality of points on thedisplayed remaining route and the selection of a route reconfigurationaffordance ra. Specifically, the evacuation guidance application 100 amay extract evacuation route information that does not include ashooting point on a selected image from among a plurality of evacuationroute information and display an evacuation route cp with a highpriority among the extracted evacuation routes. Therefore, if the userdetermines that a particular point is inappropriate as a movement routewhile checking the images for a plurality of points on the displayedremaining route, the user may reconfigure an evacuation route detouringthe corresponding point by selecting the route reconfigurationaffordance ra that has selected the image of the corresponding point andevacuate by detouring through the reconfigured evacuation route.

FIG. 9 illustrates a system according to an embodiment for describing amethod for a server to receive image data from a plurality of mobiledevices and analyze environmental state information.

Referring to FIG. 9 , the server 200 may receive image data captured bya plurality of users moving on the same evacuation route. The server 200may periodically receive image data captured at each of a plurality ofpoints on the evacuation route. After receiving image data captured atone of the plurality of points multiple times, the server 200 mayanalyze the plurality of images to detect environmental stateinformation of the corresponding point. The environmental stateinformation may include information on whether or not there is a fire atthe corresponding point and the density of people. However, the presentdisclosure is not limited to the specific example and may furtherinclude information used to determine whether it is appropriate for anevacuee to move to the corresponding point.

To determine the occurrence of fire at a particular point by analyzing aplurality of images, the server 200 may extract the features of smokeand flames using the Convolutional Neural Network (CNN). However, thepresent disclosure is not limited to the specific example above and mayuse various methods as long as they are capable of analyzing images anddetermining the occurrence of fire at a point on the image and whetherpeople may move considering the fire. Also, the server 200 may analyze aplurality of images using a deep-learning network to detect the densityof people at a point on the image.

As described above, when the server 200 determines that an analyzedpoint is an avoidance point inappropriate or impossible for people tomove through, the server may transmit a request for reconfiguring anevacuation route, location information of the avoidance point, and imagedata capturing the avoidance point to the mobile device(s) displayingthe evacuation route to which the avoidance point, a point determined asbeing inappropriate or impossible to move through, belongs among mobiledevices of a plurality of users.

FIG. 10 illustrates an evacuation guidance interface related todisplaying avoidance points and guidance tags for reconfiguring a route.

Referring to FIG. 10 , the mobile device 100 may display a receivedavoidance point dp on the map display area 112. In addition, the mobiledevice 100 may display the image dpi of the received avoidance point anddisplay a guidance tag (for example, “There is a fire in the displayedarea. Please reconfigure an evacuation route.” as shown in the figure).

The user checks the image dpi at the displayed avoidance point todetermine whether it is possible to pass the point designated as anavoidance point considering the user's physical condition. If the userdetermines that it is impossible to pass through the avoidance point,the user may evacuate through an alternative route obtained byreconfiguring the evacuation route.

Referring again to FIG. 1 , in various embodiments, at least oneevacuation guidance assisting device 300 may be installed on theevacuation route.

The evacuation guidance assisting device 300 may be installed on theceiling or passageways inside a building and guide people to anemergency exit or an escape place in a disaster situation such as fireor earthquake.

The evacuation guidance assisting device 300 may be paired with at leastone mobile device 100 within a preconfigured radius. When the userarrives at a location adjacent to the evacuation guidance assistingdevice 300 while moving along the evacuation route, the evacuationguidance assisting device 300 may be automatically paired with themobile device 100 carried by the user. The evacuation guidance assistingdevice 300 may receive specific information related to the type ofdisaster, the user's exercise ability, and personal conditions from thepaired mobile device 100. For example, if the user is hearing-impaired,the evacuation guidance assisting device 300 may output visualinformation related to movements along the user's evacuation route orspecific actions to be taken due to a disaster situation while the useris moving along the route. Also, if the user is a child under a specificage, the evacuation guidance assisting device 300 may display text orimages related to movements along the evacuation route or specificactions to be taken due to a disaster situation while the child ismoving along the route in a form easy for children to understand.

The evacuation guidance assisting device 300, being paired with theuser's mobile device 100 located within a preconfigured radius, mayprovide messages suitable for the user's conditions and currentsituation considering the type of disaster, the user's exercise ability,and the user's conditions, thereby enabling the user to evacuate safelyand quickly in a disaster situation.

FIG. 11 illustrates an evacuation guidance interface for describing amethod for detecting a state in which an emergency item has been removedand removing location information of the emergency item from anevacuation route.

Referring to FIGS. 5 and 11 , if the evacuation guidance interface 110determines that the mobile device 100 is located at a point of a routematching the location where an emergency item is disposed, theevacuation guidance interface 100 may analyze captured images todetermine whether the emergency item is detected in the captured images.

Suppose a user has used an emergency item disposed on an evacuationroute. In this case, the evacuation guidance interface of a mobiledevice 100 of a person following the user determines that the mobiledevice is located on one point of the route matching the location wherethe emergency item is disposed and analyzes captured images. However,since the emergency item has already been used, the emergency item isnot detected in the captured images. In this case, the correspondingmobile device may transmit the use completion information of theemergency item and the location of the used emergency item to the server200. When the server 200 receives the use completion information of theemergency item and the location information of the used emergency item,the server 200 may transmit information of the used emergency item andlocation information of the used emergency item to the mobile devices ofall occupants communicating with the server 200.

Among the mobile devices that have received the information of the usedemergency item and the location information of the used emergency item,mobile devices displaying the evacuation route to which the location ofthe used emergency item belongs may update the location information ofthe emergency item by removing an icon esi of the emergency item fromthe evacuation route, which is indicated by a dotted circle on the mapdisplay area 112 of FIG. 11 .

FIGS. 12 to 14 is a flow diagram illustrating execution of a real-timedisaster situation notification mode and a disaster situation batteryconsumption control mode according to a method for guiding an evacuationroute for occupants using augmented reality of mobile devices. FIGS. 15and 16 illustrate an evacuation guidance interface according toexecution of a real-time disaster situation notification mode.

Referring to FIG. 12 , in a method for guiding an evacuation route foroccupants using augmented reality of mobile devices, a method forexecuting a real-time disaster situation notification mode and adisaster situation battery consumption control mode S100 may include thefollowing steps. First, the mobile device 100 may execute S110 anevacuation guidance application 100 a in response to a request toexecute the evacuation guidance application 100 a and determine whetherdisplaying an evacuation route p has been completed S120. When themobile device 100 determines that displaying the evacuation route p hasbeen completed, the mobile device 100 may determine periodically whethera user has moved a predetermined distance along the evacuation route pbased on the location information of the mobile device 100 S130. Whenthe mobile device 100 determines that the user has moved thepredetermined distance along the evacuation route p, the mobile device100 may calculate an average movement speed due to moving thepredetermined distance and calculate the amount of battery consumptionof the mobile device 100 consumed while the user moves the predetermineddistance S140. The mobile device 100 may calculate an estimated amountof battery consumption due to movement along the remaining route basedon the average movement speed due to moving the predetermined distanceand the amount of battery consumption of the mobile device 100 consumedwhile the user moves the predetermined distance S140. In other words,the mobile device 100 may calculate an estimate of the amount of batteryconsumption due to moving the remaining route on the evacuation route p.

Next, the mobile device 100 may determine whether an estimate of theamount of battery consumption due to moving the remaining route exceedsthe amount of current battery charge S150.

When the estimate of the amount of battery consumption due to moving theremaining route exceeds the amount of current battery charge, the mobiledevice 100 may return to the S130 step in which the mobile device 100determines whether a user has moved a predetermined distance along theevacuation route p from the user's current location.

While the user moves the evacuation route p, the process above isrepeated periodically. Therefore, whether the amount of battery chargeof the mobile device 100 is sufficient may be monitored to maintain themobile device 100 to be in an active state until the mobile device 100arrives at an evacuation place.

When the mobile device 100 determines that an estimate of the amount ofbattery consumption due to moving the remaining route is less than theamount of current battery charge, the mobile device 100 may execute thereal-time disaster situation notification mode and the disastersituation battery consumption control mode S160, S170.

Referring to FIGS. 13, 15, and 16 , each of the following steps may beexecuted in the real-time disaster situation notification mode S160 ofthe mobile device 100.

When a preconfigured application of a first group is run in thebackground mode, the mobile device 100 may reduce the amount of batteryconsumption by forcing the application of the first group to beterminated S161. Here, the application of the first group means anadditional application required to operate the evacuation guidanceapplication 100 a and the remaining applications except for a messengerapplication.

The mobile device 100 may execute an application of a second group andextract information on main conversation participants who have conducteda conversation for a predetermined period S162. Here, the application ofthe second group may be a messenger application having use records for apredetermined period from a current time.

The mobile device 100 extracts information on a plurality of mainconversation participants and expands the range of selection ofappropriate conversation participants to whom the user provides anotification about a current disaster situation.

As shown in FIG. 15 , the mobile device 100 superimposes icons (m.app)of applications in the second group on the image display area 111 anddisplays main conversation participant affordances (app1.c.u1,app1.c.u2, app1.c.u3, app2.c.u1, app2.c.u2, app2.c.u3) extracted withinthe messenger applications corresponding to the respective icons(m.app). The mobile device 100 may determine whether one of the mainconversation participant affordances (app1.c.u1, app1.c.u2, app1.c.u3,app2.c.u1, app2.c.u2, app2.c.u3) displayed is selected (for example, atouch input) S164. In response to one selection from the mainconversation participant affordances (app1.c.u1, app1.c.u2, app1.c.u3,app2.c.u1, app2.c.u2, app2.c.u3) displayed, the mobile device 100 mayforce the remaining messenger applications to be terminated except forthe messenger application corresponding to the selected mainconversation participant affordance app1.c.u2 S165. The mobile device100 may transmit map information displayed on the map display area 112,evacuation route information, and the user's current location to themain conversation participant selected through the messenger applicationS166.

The mobile device 100 may determine whether a response is received fromthe selected main conversation participant within a predetermined periodS167. Suppose the mobile device 100 receives no response from theselected main conversation participant within the predetermined period.In that case, the mobile device 100 may return to the S162 step toexecute the application of the second group and extract information onthe remaining main conversation participants except for the selectedconversation participant. Suppose the mobile device 100 receives aresponse from the selected main conversation participant within thepredetermined period. Then the mobile device 100 displays a conversationmode window c.w for the selected main conversation participant bysuperimposing the window on the image display area 111 as shown in FIG.16 and executes the conversation mode with the selected mainconversation participant, displaying message information from theselected main conversation participant S168.

Referring to FIG. 14 , the mobile device 100 may determine S171 whetherthe user's biometric information is detected according to the executionof the disaster situation battery consumption control mode S170. Here,the biometric information may be the user's face information or theuser's iris information registered in advance for unlocking the mobiledevice 100 using a front camera device of the mobile device 100. Whenthe user's biometric information is detected, the mobile device 100 maymaintain the turn-on state of the display S173. In addition, the mobiledevice 100 may periodically transmit a picture of the user captured withthe front camera and a picture of the front scene to the selected mainconversation participant S174. If the user's biometric information isnot detected, the mobile device 100 may turn off the display to reducebattery consumption and provide an audio guide of evacuationinformation. In some embodiments, the mobile device 100 may measure thelevel of external noise and adjust the output volume of the audio guideof the evacuation information. The mobile device 100 continuouslydetects the user's biometric information S171 to keep the display turnedon only when the user looks at the mobile device 100. When the user doesnot look at the mobile device 100 during the evacuation, the mobiledevice 100 may turn off the display to reduce battery consumption andprovide an audio guide of evacuation information.

The embodiments of the present disclosure described above may beimplemented in the form of program commands which may be executedthrough various computer elements and recorded in a computer-readablerecording medium. The computer-readable recording medium may includeprogram commands, data files, and data structures separately or incombination thereof. The program commands recorded in the medium may bethose designed and configured specifically for the present disclosure ormay be those commonly available for those skilled in the field ofcomputer software. Examples of a computer-readable recoding medium mayinclude magnetic media such as hard-disks, floppy disks, and magnetictapes; optical media such as CD-ROMs and DVDs; and hardware devicesspecially designed to store and execute program commands such as a ROM,a RAM, and a flash memory. Examples of program commands include not onlymachine codes such as those generated by a compiler but also high-levellanguage codes which may be executed by a computer using an interpreterand the like. The hardware device may be configured to be operated byone or more software modules to perform the operations of the presentdisclosure, and vice versa.

Specific implementations of the present disclosure are embodiments,which does not limit the technical scope of the present disclosure inany other conceivable way. For the clarity of the specification,descriptions of conventional electronic structures, control systems,software, and other functional aspects of the systems may be omitted.Also, connection of lines between constituting elements shown in thefigure or connecting members illustrate functional connections and/orphysical or circuit connections, which may be replaceable in an actualdevice or represented by additional, various functional, physical, orcircuit connections. Also, unless explicitly stated otherwise,“essential” or “important” elements may not necessarily refer toconstituting elements needed for application of the present disclosure.

Also, although detailed descriptions of the present disclosure have beengiven with reference to preferred embodiments of the present disclosure,it should be understood by those skilled in the corresponding technicalfield or by those having common knowledge in the corresponding technicalfield that the present disclosure may be modified and changed in variousways without departing from the technical principles and scope specifiedin the appended claims. Therefore, the technical scope of the presentdisclosure is not limited to the specifications provided in the detaileddescriptions of the present disclosure but has to be defined by theappended claims.

DETAILED DESCRIPTION OF MAIN ELEMENTS

-   10: Evacuation guidance system-   20: Mobile terminal-   30: Smart glasses-   100: Mobile device-   100 a: Evacuation guidance application-   110: Evacuation guidance interface-   111: Image display area-   112: Map display area-   200: Server-   300: Evacuation guidance assisting device

What is claimed is:
 1. A system for guiding an evacuation route usingaugmented reality of mobile devices, the system comprising: anoccupant's mobile device installed with an evacuation guidanceapplication for showing an evacuation route from a current location ofthe occupant to an evacuation point in the occurrence of a disaster; anda server providing a plurality of evacuation route information to themobile device, wherein the mobile device executes the evacuationguidance application to display an evacuation guidance interface,displays a captured landscape image on an image display area of theevacuation guidance interface, displays one of the plurality ofevacuation routes set based on specific information related to the typeof disaster, the user's exercise ability, and personal conditions on amap display area of the evacuation guidance interface and displays guideinformation for displaying a movement direction from a current locationalong an evacuation route indicated on the map display area bysuperimposing the guide information on the image display area togetherwith the landscape image.
 2. The system of claim 1, wherein the serverprioritizes the plurality of evacuation routes based on specificinformation related to the type of disaster, the user's exerciseability, and personal conditions; and the mobile device displays one ofthe evacuation routes selected based on the priorities of the pluralityof evacuation routes received from the server on the map display area.3. The system of claim 1, wherein the mobile device receives locationinformation of at least one emergency item related to the type ofdisaster from the server and displays the location information of anevacuation route on the map display area.
 4. The system of claim 3,wherein, when the mobile device is located at a point on an evacuationroute displayed on the map display area that matches the location atwhich the emergency item is disposed, the mobile device analyzes acaptured image to determine whether an emergency item is detected withinthe captured image; and displays an indication to facilitateidentification of the emergency item when the emergency item is detectedwithin the captured image and displays a tag including information onhow to use the detected emergency item.
 5. The system of claim 3,wherein, when the mobile device is located at a point on an evacuationroute displayed on the map display area that matches the location wherethe emergency item is disposed, the mobile device analyzes a capturedimage to determine whether an emergency item is detected in the capturedimage; and transmits use completion information of the emergency itemand location information of the emergency item completed for use to theserver when the emergency item is not detected in the captured image;and to remove the location of an emergency item completed for usedisplayed on the evacuation route, the server transmits information ofthe emergency item completed for use and location information of theemergency item completed for use to the mobile devices of all occupantscommunicating with the server.
 6. The system of claim 1, wherein themobile device receives waypoint state images, which are images capturedby other mobile devices at a plurality of different points on theremaining route from a current location to an evacuation point, from theserver and displays the received images.
 7. The system of claim 6,wherein the mobile device reconfigures an evacuation route that detoursa point on a waypoint state image selected in response to a selection ofthe displayed waypoint state image and a route reconfigurationaffordance and displays the reconfigured evacuation route on the mapdisplay area.
 8. The system of claim 6, wherein waypoint state imagesdisplayed on the mobile device are updated periodically according to alocation change of the mobile device.
 9. The system of claim 8, whereinthe number of waypoint state images transmitted from the server to theuser's mobile device is determined based on the remaining distance froma current location of the mobile device to the evacuation point, thenumber of other users' mobile devices on the remaining route from thecurrent location to the evacuation point, and separation distances amongpositions of other users' mobile devices.
 10. The system of claim 1,wherein the server receives captured image data for the same pointphotographed by each of a plurality of users moving along the sameevacuation route, analyzes the plurality of captured image data todetect environmental state information of the same point and, if thesame point is determined as an avoidance point through which movement isnot appropriate, transmits a request for reconfiguring an evacuationroute, location information of the avoidance point, and images capturingthe avoidance point to the mobile devices of a plurality of users movingalong the same evacuation route; and the mobile device receiving therequest for reconfiguring an evacuation route, location information ofthe avoidance point, and images capturing the avoidance point displaysthe avoidance point, the images capturing the avoidance point, and aguidance tag requesting reconfiguration of an evacuation route anddisplays an evacuation route reconfigured to avoid a point on the imagecapturing a displayed avoidance point in response to the selection of aroute reconfiguration affordance.
 11. The system of claim 1, wherein themobile device calculates an average movement speed over a preconfigureddistance and an amount of battery consumption of the mobile deviceconsumed while moving the preconfigured distance by determining whetherthe movement of the preconfigured distance along the evacuation routehas been completed, determines whether an estimated amount of batteryconsumption required for movement along the remaining route exceeds theamount of current battery charge, and executes a real-time disastersituation notification mode and a disaster situation battery consumptioncontrol mode if the estimate does not exceed the amount of currentbattery charge; the mobile device displays a messenger applicationhaving use records for a predetermined period and displays informationon main conversation participants who have conducted a conversation fora preconfigured time by extracting the information in the messengerapplication, in response to selection of the displayed information onmain conversation participants who have conducted a conversation,transmits real-time evacuation route information and locationinformation of a current user to the selected main conversationparticipants, and executes a conversation mode with the selected mainconversation participants if a response is received from the selectedmain conversation participants within a preconfigured time; and themobile device determines whether biometric information of apre-registered user is detected, maintains a display of the mobiledevice to be in a turned-on state when the biometric information isdetected, transmits an image photographing the user and captured imageof the front scene to the selected main conversation participants, andturns off the display and provides an audio guide of evacuationinformation while the biometric information is not detected.